Pixel signal processing apparatus and pixel signal processing method

ABSTRACT

A regression analysis is carried out ( 8 ) using pixel signals having a K-th spectral characteristic as the explanatory variable and pixel signals having an L-th spectral characteristic as the purpose variable in a plurality of pixel positions in an area neighboring a pixel of interest to obtain a pixel signal having the L-th spectral characteristic ( 9 ). Pixel signals obtained by low-pass filtering ( 7   a - 7   c ) of the output signals of an imaging device may be used as the explanatory variable and the purpose variable. The occurrence of false colors is thereby reduced when, in a group of pixel signals from pixels arrayed on a two-dimensional plane, each pixel having one of a plurality of spectral characteristics, the missing colors at each pixel position are obtained by interpolation.

FIELD OF THE INVENTION

The present invention relates to a pixel signal processing apparatus anda pixel signal processing method, more particularly to a pixel signalprocessing apparatus and method that, based on a group of pixel signalsof pixels obtained from a two-dimensional plane array of pixel positionseach having one of a plurality of spectral characteristics, generates,for a pixel position of interest at which there is a pixel signal havingone of those spectral characteristics, pixel signals of the otherspectral characteristics.

This type of pixel signal processing apparatus is used as part of acolor imaging apparatus that also includes a color imaging device havingmultiple types of photoelectric conversion elements arrayed on atwo-dimensional plane, for example, a Bayer array of imaging elements,each having one of a plurality of spectral characteristics such as thethree spectral characteristics or colors red (R), green (G), and blue(B); the apparatus is used to interpolate color signals with spectralcharacteristics that are lacking at each pixel position in the pixelsignals output from the image elements.

BACKGROUND ART

In conventional imaging apparatus having imaging elements with a Bayerarray of color filters of the three primary colors red, green, and blue,to increase the sense of resolution, the output signal of each pixel isreplaced with a mean value based on the local distribution of outputsignals for each color, thereby employing an interpolation method basedon an assumed linear similarity between the known color geometry and themissing color geometry, as shown, for example, in Patent Document 1below.

-   Patent Document 1: Japanese Patent Application Publication No.    2000-197512 (paragraphs 0048 to 0049, FIG. 7.)

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

A problem in this conventional method, in which interpolation is carriedout on the basis of correlation between the color signals in a localarea, is the occurrence of false colors due to misapprehension of thecorrelation relationship between colors. An example of this problemoccurs in the neighborhood of a site of a sudden change in the outputsignal, that is, near a boundary between one color and another color:the generated signal levels deviate greatly from the curve of signalvariation, giving rise to black or white smears or false colors thatwere not present in the original image.

Means of Solution of the Problems

In a pixel signal processing apparatus that, given a group of pixelsignals from pixels arrayed on a two-dimensional plane, each pixelhaving one of a first to an N-th spectral characteristic, generates apixel signal having an L-th spectral characteristic at a first pixelposition of interest where there is a pixel signal having a K-thspectral characteristic (K and L being different integers between 1 andN, inclusive), an imaging apparatus according to the present inventionhas:

regression analysis means for performing a regression analysis in aplurality of pixel positions in an area neighboring the first pixelposition of interest, using the pixel signals having the K-th spectralcharacteristic as the explanatory variable and the pixel signals havingthe L-th spectral characteristic as the purpose variable, to calculate aregression equation expressing a correlation of the pixel signal havingthe K-th spectral characteristic with the pixel signal having the L-thspectral characteristic; and

a calculating means for determining the pixel signal having the L-thspectral characteristic at the pixel position of interest by applying aconversion formula based on the regression equation to the pixel signalhaving the K-th spectral characteristic at the pixel position ofinterest.

Effect of the Invention

A pixel signal processing apparatus according to this invention, beingstructured as above, can calculate a generated signal fitting thecorrelation relationship between the pixel signals with the K-th andL-th spectral characteristics at pixel positions in the area neighboringthe first pixel position of interest. Even when the first pixel ofinterest is located near a color boundary, accordingly, it is notdirectly affected by the output signals of the pixels at the colorboundary, because the L-th generated signal is calculated from theregression equation indicating the correlation relationship between thepixels in the area neighboring the first pixel position of interest. Anycorrelation relationship can be dealt with, because the correlationrelationship is expressed mathematically. The occurrence of false colorssuch as the black or white smears seen with conventional methods cantherefore be reduced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the structure of an imaging devicesuch as a digital still camera equipped with the pixel signal processingapparatus of a first embodiment of this invention.

FIG. 2 illustrates a Bayer array of color filters of the three primarycolors red, green, and blue.

FIG. 3 schematically illustrates the two-dimensional arrangement ofoutput signals in a two-dimensional imaging device with a Bayer array ofcolor filters.

FIGS. 4(a), 4(b), and 4(c) separately illustrate the two-dimensionalarrangement of each color in FIG. 3.

FIG. 5 is a flowchart illustrating the interpolation procedure in thethree embodiments of the invention described herein.

FIG. 6 is a flowchart illustrating the procedure for generating a greensignal at a pixel having a red output signal in the first embodiment ofthe invention.

FIG. 7 schematically illustrates a regression line obtained from a setof nine data points representing output from the red and green low-passfilters.

FIG. 8 is a flowchart illustrating the calculation of the slope andintercepts of the regression line in the three embodiments of theinvention.

FIG. 9 shows exemplary values of the output signal obtained byanalog-to-digital conversion of the output of the imaging device.

FIG. 10 illustrates low-pass filter outputs calculated from the outputsignal values in FIG. 9 in the first embodiment of the invention.

FIG. 11 illustrates a regression line obtained from the low-pass filteroutputs in FIG. 10 in the first embodiment of the invention.

FIG. 12 is a block diagram showing the structure of an imaging devicesuch as a digital still camera equipped with a pixel signal processingapparatus according to a second embodiment of the invention.

FIG. 13 is a flowchart illustrating the procedure for generating a greensignal at a pixel having a red output signal in the second embodiment ofthe invention.

FIG. 14 is a flowchart illustrating the procedure by which theregression analysis means evaluates pixel similarity in the secondembodiment of the invention.

FIG. 15 illustrates the arrangement of pixels referred to in generatingthe missing green color when the pixel of interest has a red outputsignal in the second embodiment of the invention.

FIG. 16 is a flowchart illustrating the procedure for generating a redsignal at a pixel having a green output signal in the second embodimentof the invention.

FIG. 17 illustrates the arrangement of pixels referred to in generatingthe missing red and blue colors when the pixel of interest has a greenoutput signal in the second embodiment of the invention.

FIG. 18 is a flowchart illustrating the procedure for generating a bluesignal at a pixel having a red output signal in the second embodiment ofthe invention.

FIG. 19 illustrates the arrangement of pixels referred to in generatingthe missing blue color when the pixel of interest has a red outputsignal in the second embodiment of the invention.

FIG. 20 is a block diagram showing the structure of an imaging devicesuch as a digital still camera equipped with a pixel signal processingapparatus according to a third embodiment of the invention.

FIG. 21 is a flowchart illustrating the procedure for generating a redsignal at a pixel having a green output signal in the third embodimentof the invention.

FIG. 22 illustrates the arrangement of pixels referred to in generatingthe missing red and blue colors when the pixel of interest has a greenoutput signal in the third embodiment of the invention.

FIG. 23 is a flowchart illustrating the procedure for generating a bluesignal at a pixel having a red output signal in the third embodiment ofthe invention.

FIG. 24 illustrates the arrangement of pixels referred to in generatingthe missing blue color when the pixel of interest has a red outputsignal in the third embodiment of the invention.

EXPLANATION OF REFERENCE CHARACTERS

2 two-dimensional imaging device, 7 a-7 c low-pass filters, 8, 10, 12regression analysis means, 9, 13 calculating means

BEST MODE OF PRACTICING THE INVENTION

Embodiments of the invention will now be described with reference to theattached drawings. The embodiments described below are suitable for usein a digital still camera, but applications of this invention are notlimited thereto.

First Embodiment

FIG. 1 is a block diagram showing the structure of an imaging devicehaving pixel signal processing apparatus according to the firstembodiment of the invention. Light incident on a lens 1 is focused on,for example, a two-dimensional imaging device 2 with color filtershaving spectral characteristics corresponding to the red (R), green (G),and blue (B) primary colors arranged in a Bayer array as shown in FIG.2.

The two-dimensional imaging device 2 carries out photoelectricconversion of the incident light and outputs an analog signal at a levelaccording to the intensity of the incident light. The analog signal isconverted to a digital signal by an analog-to-digital (A/D) converter 3,then output and stored in a frame memory 4.

The pixel signals stored in the frame memory 4 are demultiplexed by ademultiplexer 5 into the red, green, and blue colors and storedseparately in two-dimensional memories 6 a, 6 b, 6 c. Red signals arestored in two-dimensional memory 6 a, green signals in two-dimensionalmemory 6 b, and blue signals in two-dimensional memory 6 c. Low-passfilters (LPFs) 7 a, 7 b, 7 c are provided for the two-dimensionalmemories 6 a, 6 b, 6 c to perform low-pass filtering of the pixelsignals read from the memories 6 a, 6 b, 6 c and output the results.

Because the red, green, and blue color filters of the two-dimensionalimaging device 2 are disposed at the positions of the correspondingpixels in, for example, the Bayer array shown in FIG. 2, a pixel signalof only one color is obtained from each pixel position; the pixelsignals of the other two colors are not obtained. In other words, ateach pixel position a pixel signal of only one color is present; thepixel signals of the other colors are absent.

The colors of the pixels signals that are absent will be referred tobelow as ‘missing colors’. In the output of the imaging device 2, forexample, the missing colors at a pixel position where a red pixel signalis present are green and blue.

In the present invention, the regression analysis means 8 andcalculating means 9 use the pixel signals output from the low-passfilters 7 a-7 c to obtain pixel signals of the missing colors at eachpixel position by interpolation.

The interpolation procedure includes the following six processes.

-   (P1) A process for determining the green pixel signal at pixel    positions where a red pixel signal is present-   (P2) A process for determining the green pixel signal at pixel    positions where a blue pixel signal is present-   (P3) A process for determining the red pixel signal at pixel    positions where a green pixel signal is present-   (P4) A process for determining the blue pixel signal at pixel    positions where a green pixel signal is present-   (P5) A process for determining the blue pixel signal at pixel    positions where a red pixel signal is present-   (P6) A process for determining the red pixel signal at pixel    positions where a blue pixel signal is present

These six processes can be generalized as:

-   -   Calculate L signal (L=R, G, or B) at    -   K pixel position (K=R, G, or B, K≠L)        Each of these six processes is carried out at every pixel        position on the screen (in one frame).

The six processes above may be carried out sequentially. When the sixprocesses above are carried out at each pixel position (which thenbecomes the pixel position of interest) the regression analysis means 8receives the pixel signals of the K-th and L-th colors in an areaneighboring the pixel position of interest (an area including the firstpixel of interest and its surrounding pixels (within a predetermineddistance of the pixel of interest)) from the relevant low-pass filters(two of the filters 7 a-7 c) and calculates a regression equationexpressing the correlation between the received pixel signals.

The calculating means 9 uses the constants in the regression equationcalculated by the regression analysis means 8 and the pixel signal ofthe K-th color at the pixel position of interest (the output of theimaging device 2), which is stored in the frame memory 4, to calculatethe pixel signal of the L-th color at the pixel position of interest,and stores the pixel signal resulting from the calculation in the resultmemory 14. The pixel signal resulting from the above calculation will bereferred to below as the ‘generated pixel signal’ or simply the‘generated signal’. The pixel signals obtained as results of filteringin the low-pass filters 7 a-7 c may be referred to simply as ‘low-passfilter outputs’. The pixel signals before the low-pass filtering, thatis, the pixel signals obtained by A/D conversion of the outputs of theimaging device, may be referred to simply as ‘output signals’. As A/Dconversion only changes the form of the signals, the pixel signalsstored in the frame memory 4 may also be referred to as the pixelsignals output from the imaging device 2, or the output signals of theimaging device.

When all six processing steps above have been completed, pixel signalsare present for the missing colors at all pixel positions on the screen.The combination of the generated signals stored in the result memory 14and the output signals stored in the frame memory 4 forms a complete setof pixel signals for all colors (red, green, blue) in all pixelpositions on one screen. The calculating means 9 outputs this set ofpixel signals, formed by the combination of the generated signals storedin the result memory 14 and the output signals stored in the framememory 4, as an RGB color signal.

The embodiment will be described in detail below.

FIG. 3 schematically illustrates a two-dimensional arrangement of thepixel signals obtained by A/D conversion of the outputs of the imagingdevice 2. Each grid cell in the drawing indicates a pixel position. Theletter R, G, or B and the numerals in parentheses in the cells indicatethe color of the pixel signal and the coordinates of the pixel position(l=row, m=column).

As shown in FIG. 3, in the output signals, at each pixel position thereis a pixel signal of only one color; there are no pixel signals of theother colors at the same pixel position. In other words, the pixelsignals are arranged so that different colors never occupy the samepixel position.

The outputs of the A/D converter 3 are stored in the frame memory 4 asdescribed above, and the signals read from the frame memory 4 aredemultiplexed by the demultiplexer 5 and stored, color by color, intothe two-dimensional memories 6 a, 6 b, 6 c.

FIGS. 4(a) to 4(c) schematically illustrate the pixel signals stored inthe two-dimensional memories 6 a to 6 c, and their positions.

As shown in the drawings, the output signals of each color fail tooccupy all the pixel positions: for example, only one in four pixelpositions is occupied by a red output signal, leaving three pixelpositions for pixel signals of the other colors; only one in two pixelpositions is occupied by a green color output signal, leaving one pixelposition for a pixel signal of another color; only one in four pixelpositions is occupied by a blue color output signal, leaving three pixelpositions for pixel signals of the other colors.

The pixel signal processing apparatus of the invention obtains the pixelsignals of all the missing colors at each pixel position byinterpolation.

An interpolation process for generating the missing color pixel signalsat each pixel position, performed by the low-pass filters 7 a to 7 b,regression analysis means 8, and calculating means 9, will be describedin more detail below. FIG. 5 is a flowchart illustrating theinterpolation procedure.

In the following description, when a pixel signal is present at acertain pixel position in the output of the imaging device 2, this maybe expressed by saying that the pixel ‘has an output signal’, and when apixel signal is absent at the pixel position, this may be expressed bysaying that the pixel ‘has no output signal’.

First, green signals are generated at pixel positions having red outputsignals (step ST9). Next, green signals are generated at pixel positionshaving blue output signals (step ST10). Next, red signals are generatedat pixel positions having green output signals (step ST11). Next, bluesignals are generated at pixel positions having green output signals(step ST12). Next, blue signals are generated at pixel positions havingred output signals (step ST13). Finally, red signals are generated atpixel positions having blue output signals (step ST14).

Next, assuming that a pixel having a red output signal is selected asthe pixel of interest, the process (step ST9) for generating the greensignal, which is one of the missing colors of the pixel of interest,will be described in detail. FIG. 6 is a flowchart illustrating theprocedure of step ST9.

Suppose, for example, that the pixel at position (2, 2) in FIG. 3,having a red output signal, is selected as the pixel of interest (stepST16). The notation ‘R(2, 2)’ may be used as shown in step ST16 in FIG.6 to indicate that the pixel at position (2, 2) has a red (R) outputsignal. Similar notation will be applied below to pixels having outputsignals of the other colors.

When the pixel at the pixel position (2, 2) in FIG. 3, having a redoutput signal, is selected as the pixel of interest as described above,the regression analysis means 8 receives data from the low-pass filters7 a, 7 b, the data including the red and green low-pass filter outputsat the pixel positions in an area neighboring the pixel position (2, 2)of interest in FIG. 3, that is, at the pixel position (2, 2) of interestand its eight surrounding pixel positions (1, 1), (1, 2), (1, 3), (2,1), (2, 3), (3, 1), (3, 2), (3, 3).

In the example shown in FIG. 3, the surrounding pixel positions aredefined as the pixel positions within a distance of one pixel from thepixel of interest in the horizontal and vertical directions.

The low-pass filter outputs may be obtained as a mean value of theoutput signals of the pixels having the same color that are presentwithin a predetermined area. Specifically, when the mean value of theoutput signals of the pixels of the same color included within a 3×3area is defined as the low-pass filter output at the center position inthe area, the mean value of the output signals of the four red pixelsincluded in the 3×3 region, shown by thick lines in FIG. 4(a), becomesthe low-pass filter output for the red color at the center position(l=1, m=1) of the 3×3 area. Accordingly, the red low-pass filter outputsat the nine pixel positions (1, 1), (1, 2), (1, 3), (2, 1), (2, 2), (2,3), (3, 1), (3, 2), (3, 3) in FIG. 3 can be obtained from the followingequations (4) to (12), where the notation R_(LPF) indicate a redlow-pass filter output.R _(LPF)(1, 1)=(R(0, 0)+R(0, 2)+R(2, 0)+R(2, 2))/4   (4)R _(LPF)(1, 2)=(R(0, 2)+R(2, 2))/2   (5)R _(LPF)(1, 3)=(R(0, 2)+R(0, 4)+R(2, 2)+R(2, 4))/4   (6)R _(LPF)(2, 1)=(R(2, 0)+R(2, 2))/2   (7)R _(LPF)(2, 2)=R(2, 2)   (8)R _(LPF)(2, 3)=(R(2, 2)+R(2, 4))/2   (9)R _(LPF)(3, 1)=(R(2, 0)+R(2, 2)+R(4, 0)+R(4, 2))/4   (10)R _(LPF)( 3, 2 )=(R(2, 2)+R(4, 2))/2   (11)R _(LPF)(3, 3)=(R(2, 2)+R(2, 4)+R(4, 2)+R(4, 4))/4   (12)

Similarly, the green low-pass filter outputs at the nine pixel positions(1, 1), (1, 2), (1, 3), (2, 1), (2, 2), (2, 3), (3, 1), (3, 2), (3, 3)in FIG. 3 can be obtained by the following equations (13) to (21), wherethe character G_(LPF) indicates a green low-pass filter output.G _(LPF)(1, 1)=(G(0, 1)+G(1, 0)+G(1, 2)+G(2, 1))/4   (13)G _(LPF)(1, 2)=(G(0, 1)+G(0, 3)+G(1, 2)+G(2, 1)+G(2, 3))/5   (14)G _(LPF)(1, 3)=(G(0, 3)+G(1, 2)+G(1, 4)+G(2, 3))/4   (15)G _(LPF)(2, 1)=(G(1, 0)+G(1, 2)+G(2, 1)+G(3, 0)+G(3, 2))/5   (16)G _(LPF)(2, 2)=(G(1, 2)+G(2, 1)+G(2, 3)+G(3, 2))/4   (17)G _(LPF)(2, 3)=(G(1, 2)+G(1, 4)+G(2, 3)+G(3, 2)+G(3, 4))/5   (18)G _(LPF)(3, 1)=(G(2, 1)+G(3, 0)+G(3, 2)+G(4, 1))/4   (19)G _(LPF)(3, 2)=(G(2, 1)+G(2, 3)+G(3, 2)+G(4, 1)+G(4, 3))/5   (20)G _(LPF)(3, 3)=(G(2, 3)+G(3, 2)+G(3, 4)+G(4, 3))/4   (21)

The regression analysis means 8 uses the red low-pass filter outputsR_(LPF) obtained in equations (4) to (12) as an explanatory variablee(i) (step ST17) and the green low-pass filter outputs G_(LPF) obtainedin equations (13) to (21) as a purpose variable p(i) (step ST18), andperforms a regression analysis to calculate a regression linerepresented by equation (22) below (step ST19).G _(LPF) =a×R _(LPF) +b   (22)

An example of a regression line is shown in FIG. 7. The regressionequation expressing this type of regression line is obtained by theleast squares method using the low-pass filter outputs G_(LPF) as thepurpose variable and the low-pass filter outputs R_(LPF) as theexplanatory variable.

FIG. 8 is a flowchart illustrating the detailed correlation calculationprocedure performed in step ST19 to calculate the constants ‘a’ (slope)and ‘b’ (intercept) of the regression line. The explanatory variable isdenoted by e(i), the purpose variable by p(i), the total number of datapoints by N (in the above example, N=9), and the data index by i (i=1 toN).

As shown in FIG. 8, first, in the data count setting step ST1, the totalnumber of data points used for the calculation is set. Next, in themultiplication and addition step ST2, multiplication and additionoperations are performed on the explanatory variable and purposevariable to calculate parameters k1, k2, k3. Next, in the slopecalculation step ST3, the slope ‘a’ of the regression line is calculatedusing the parameters k1 and k2 obtained in step ST2. Finally, in theintercept calculation step ST4, the intercept ‘b’ of the regression lineis calculated using the parameters k2 and k3 obtained in step ST2.

The regression line thus obtained represents a correlation between thered and green colors in an area neighboring the pixel of interest, sothe red and green signals at the pixel of interest can be presumed tohave values near this regression line.

The calculating means 9 receives the constants ‘a’ and ‘b’ calculated bythe regression analysis means 8 giving the slope and intercept of theregression line and the output signal R(2, 2) of the pixel of intereststored in the frame memory 4 and performs a conversion based on equation(23) below, thereby generating the green signal g(2, 2) at the positionof the pixel of interest (step ST20). The green, red, and blue signalsgenerated as in the above example will be indicated by the lower-caseletters ‘g’, ‘r’, and ‘b’, respectively.g(2, 2)=a×R(2, 2)+b   (23)

The generated signals are stored into the result memory 14 (step ST21).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels as the pixel of interest (step ST22).Naturally, every time the pixel of interest is changed, its surroundingpixels change and the regression analysis is performed using thesedifferent pixels, so that different values of the constants ‘a’ and ‘b’are obtained for use in obtaining the generated signal for each pixel.

The process in step ST9 in FIG. 5 is thereby completed.

In each of steps ST10 to ST14, a series of processes similar to theabove is performed as in step ST9 but with different colors.

When steps ST9 to ST14 in FIG. 5 are all completed, the missing colorsof all pixels have been interpolated, so that if the output signals (thepixel signals obtained by A/D conversion of the outputs of the imagingdevice 2) and the generated signals (the pixel signals obtained byinterpolation) are combined, pixel signals of all colors are obtainedfor all pixels. That is, a color image filling one screen is obtained.The calculating means 9 accordingly combines the generated signalsstored in the result memory 14 with the output signals stored in theframe memory 4 and outputs them as RGB signals (step ST15).

In the above example, the order in which the color signals are generatedis not limited to the order shown in FIG. 5; the order can be changed.

The above interpolation method enables interpolation by generatingsignals according to regression lines expressing an arbitrarycorrelation relationship among the color signals of the pixel ofinterest and its surrounding pixels. Therefore, the image degradationproblems in the method described in Patent Document 1 above (JapanesePatent Application Publication No. 2001-197512), such as black or whitesmears occurring near color boundaries, is greatly mitigated.

When output signals having the numeric values shown in FIG. 9 areobtained from the pixels and a color boundary is present at the locationindicated by the dotted line, for example, the red and green low-passfilter outputs at the central pixel (the pixel of interest) and itssurrounding pixels become take the values shown in FIG. 10. Theregression line obtained from these values is shown in FIG. 11. Theslope and intercept of this regression line are 0.0807 and 74.591,respectively. By substituting these results and the output signal of thepixel of interest, i.e., R(2, 2)=9, into equation (23), the missinggreen signal is calculated as shown in the following equation (24).g(2, 2)=0.0807×9+74.591≈75   (24)

The value g(2, 2) thus calculated is near the green color values of thesurrounding pixels. Image degradation such as black or white smears andthe like does not occur near the color boundary.

In the above example, the low-pass filter takes the simple average ofthe output signals of surrounding pixels, but it may also take a weighedaverage.

Second Embodiment

FIG. 12 is a block diagram showing the structure of an imaging devicehaving pixel signal processing apparatus according to a secondembodiment of the invention.

The second embodiment has the same structure as the first embodimentexcept for the regression analysis means 10 and low-pass filters 7 a to7 c.

The regression analysis means 10 of the present embodiment determinesimage similarities surrounding a pixel of interest prior to thecalculation of its regression line. Specifically, for example, theregression analysis means 10 calculates differences between the outputsignals (pixel signals obtained by A/D conversion of the outputs of theimaging device 2) of two pixels and compares them, the two pixels beingselected from among pixels having output signals of the same color asthe color to be generated as one of the missing colors of the pixel ofinterest and positioned on opposite sides of the pixel of interest inthe vertical, horizontal, or diagonal direction. The regression analysismeans 10 then determines that the direction of the line connecting thetwo pixels having the smallest output signal difference is the directionin which a strong similarity exists.

From among the pixels aligned in this direction of strong similarity,the regression analysis means 10 selects a plurality of pixels havingthe same color as the output signal of the pixel of interest, andreceives the output signals of the selected pixels from the frame memory4 and the low-pass filter outputs of the selected pixels from thelow-pass filters 7 a to 7 c.

Regression analysis is then performed using the output signals of theselected pixels as an explanatory variable and the low-pass filteroutputs of the selected pixels as a purpose variable to calculate aregression line expressing a color correlation.

Next, the operation will be described. The interpolation procedurefollows the flowchart in FIG. 5 as in the first embodiment. First, theprocess in step ST9 in FIG. 5 will be described, in which a pixel havinga red output signal is selected as the pixel of interest and a greensignal is generated to supply one of its missing colors. FIG. 13 is aflowchart illustrating this procedure.

First, a pixel at a pixel position (l, m) having a red output signal isselected as the pixel of interest (ST23).

Next, the image similarities surrounding the pixel of interest aredetermined (ST24, ST25, ST26, ST27).

The same steps as steps ST24, ST25, ST26, and ST27 in FIG. 13 are alsoshown in FIG. 14. In FIG. 14, however, exemplary specific coordinatevalues are given at the pixel positions. The procedure will be describedbelow with reference to steps ST24, ST25, ST26, and ST27 in FIG. 13 andthe corresponding steps ST5, ST6, ST7, and ST8 in FIG. 14 along withFIG. 15, which shows pixels surrounding the pixel of interest.

It is assumed here that as the pixel of interest having a red outputsignal at a pixel position (l, m), the pixel at position (3, 3) in FIG.15, for example, is selected (step ST23 in FIG. 13). From the pixelsabove, below, to the left, and to the right of the pixel of interest(the pixels aligned with it in the vertical and horizontal directions),the pixels having output signals of the same color (green) as themissing color to be generated are used to compare image similarity inthe horizontal and vertical directions around the pixel of interest. Toperform this comparison, the difference values G(V), G(H) of the greenpixels present above and below and to the left and right of the pixel ofinterest are first calculated (step ST24 or ST5). Next, these differencevalues G(V), G(H) are compared to determine which is smaller and whichis larger (step ST25 or ST6), and the direction showing the smallerdifference value, either the horizontal (left-right) direction or thevertical (up-down) direction, is determined to be the direction in whicha strong similarity exists. Among the pixels aligned in the direction ofstrong similarity, the pixels neighboring the pixel of interest andhaving output signals of the same red color as the color of the pixel ofinterest are then selected (steps ST26 and ST27 or ST7 and ST8).

Specifically, if G(V)≦G(H), the vertical direction is determined to bethe direction of strong similarity, and among the pixels aligned in thevertical direction, the pixels R(l−2, m), R(l, m), R(l+2, m) havingoutput signals of the same red color as the color of the pixel ofinterest, e.g., the pixels R(1, 3), R(3, 3), and R(5, 3), are selected(step ST26 or ST7). If G(V)>G(H), the horizontal direction is determinedto be the direction of strong similarity, and among the pixels alignedin the horizontal direction, the pixels R(l, m−2), R(l, m), R(l, m+2)having output signals of the same red color as the color of the pixel ofinterest, e.g., the pixels R(3, 1), R(3, 3), and R(3, 5), are selected(step ST27 or ST8).

After the similarity has been thus determined, the regression analysismeans 10 receives the red output signals of the selected pixels from theframe memory 4 for use as an explanatory variable e(i) (i=1 to 3) (stepST28), and receives the green low-pass filter outputs of the selectedpixels from low-pass filter 7 b for use as a purpose variable p(i) (i=1to 3) (step ST29).

The low-pass filters 7 a to 7 c receive the result of the similaritydetermination, and perform low-pass filtering for each pixel positionbased only on the output signals of the pixels aligned in what has beendetermined to be the direction of strong similarity. In the abovedetermination of similarity, for example, when it is determined that thestronger similarity exists in the vertical direction, the mean value ofthe output signals of the pixels in an area measuring three pixelsvertically by one pixel horizontally with the pixel of interest at thecenter is output as a low-pass filter output. If in the abovedetermination of similarity it is determined that a stronger similarityexists in the horizontal direction, the average value of the outputsignals of the pixels in an area measuring one pixel vertically by threepixels horizontally is output as a low-pass filter output. Specifically,when it is determined that a strong similarity exists in the verticaldirection, for example, the low-pass filter output at the pixel position(1, 3) in FIG. 15 is obtained as the mean value of the output signals ofthe pixels included in the area shown by a thick line. The low-passfilter outputs of the other pixel positions (3, 3), (5, 3) are obtainedsimilarly. Accordingly, when the vertical direction is the direction ofstrong similarity, the green low-pass filter outputs of the selectedpixels are calculated by the following equations (25) to (27).G _(LPF)(1, 3)=(G(0, 3)+G(2, 3))/2   (25)G _(LPF)(3, 3)=(G(2, 3)+G(4, 3))/2   (26)G _(LPF)(5, 3)=(G(4, 3)+G(6, 3))/2   (27)

The regression analysis means 10 performs a regression analysis (stepST30) using the explanatory variable e(i) (red output signals) andpurpose variable p(i) (green low-pass filter outputs), calculates theslope and intercept values of the regression line showing thecorrelation between the red and green colors, and then calculates thegenerated signal of the green color of the pixel of interest (step ST31)as in the first embodiment. The generated signal is stored into theresult memory 14 (step ST32).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels as the pixel of interest (step ST33).

The process in step ST9 in FIG. 5 is thereby completed.

In step ST10 in FIG. 5, the above series of processes is performed as instep ST9 but with the color changed from red to blue to obtain greensignals at pixels having a blue output signal.

In step ST11 in FIG. 5, a series of processes is performed to obtain redsignals at pixels having a green output signal. These processes differfrom those in steps ST9 and ST10 as described below. Step ST11 will bedescribed below in detail with reference to FIGS. 16 and 17.

First, for example, the pixel at pixel position (3, 3) in FIG. 17,having a green output signal, is selected as the pixel of interest(ST34).

In the neighborhood of a pixel having a green output signal, pixelshaving output signals of the red missing color are aligned only in thehorizontal direction or only in the vertical direction. Therefore, theregression analysis means 10 does not determine the similaritydirection, but determines whether the pixels having the relevant colorare aligned in the horizontal or vertical direction (ST35), and selectsthe direction in which they are present (steps ST36 and ST37). When therelevant pixels are located in the horizontal direction as shown in FIG.17, for example, the pixels at pixel positions (l, m−2), (l, m), and (l,m+2), which are aligned in the horizontal direction and have green pixelsignals, are selected (ST36). When the relevant pixels are located inthe vertical direction (differing from FIG. 17), the pixels at pixelpositions (l−2, m), (l, m), and (l+2, m), which are aligned in thevertical direction and have green pixel signals, are selected (ST37).

The other processes (ST38 to ST43) are the same as the processesdescribed in steps ST28 to ST33 in FIG. 13.

That is, as when green signals are generated for pixels having a redoutput signal, as described with reference to FIG. 13, the outputsignals of the selected pixels (the signals obtained by A/D conversionof the outputs of the imaging device 2) are set as an explanatoryvariable e(i) (step ST38), the red low-pass filter outputs of theselected pixels are set as a purpose variable p(i) (step ST39), and aregression analysis is performed (step ST40). The red generated signalof the pixel of interest is then calculated from the resultingregression line and the output signal of the pixel of interest (stepST41). The generated signal is stored into the result memory 14 (stepST42).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels as the pixel of interest (step ST43).

The process in step ST11 in FIG. 5 is thereby completed.

In step ST12 in FIG. 5, a series of processes is performed as in stepST11 above but with the color changed from red to blue to obtain bluesignals at pixels having a green output signal.

In step ST13 in FIG. 5, a series of processes is performed to obtainblue signals at pixels having a red output signal. These processesdiffer from those in steps ST9 to ST12 as described below. Step ST13will be described below in detail with reference to FIGS. 18 and 19.

First, the pixel at pixel position (3, 3) in FIG. 19, having a redoutput signal, is selected as the pixel of interest (ST44).

Next, the similarities among the pixels surrounding the pixel ofinterest are determined (ST45, ST46, ST47, ST48). In this case,differing from FIG. 13, the similarities in the two diagonal directionstilted by 45 degrees with respect to the horizontal and verticaldirections are obtained (steps ST45 to ST48).

That is, from among the pixels aligned in the diagonally ascending anddescending directions passing through the pixel of interest, pixelshaving output signals of the same red color as the missing color to begenerated are used to compare image similarities in the diagonallyascending and descending directions passing through the pixel ofinterest. To perform this comparison, first, the blue pixels aligned inthe diagonally ascending direction passing through the pixel of interestand positioned on opposite sides of the pixel of interest are used tocalculate the difference value B(D1)=|B(l−1, m+1)−B(l+1, m−1)|, and theblue pixels aligned in the diagonally descending direction passingthrough the pixel of interest and positioned on opposite sides of thepixel of interest are used to calculate the difference valueB(D2)=|B(l−1, m−1)−B(l+1, m+1) (step ST45). Next, these differencevalues B(D1), B(D2) are compared to determine which is smaller and whichis larger (step ST46), and the direction (diagonally ascending ordescending direction) having the smaller difference value is determinedto be the direction of strong similarity. The pixels neighboring thepixel of interest and having output signals of the same red color as thecolor of the pixel of interest are then selected from among the pixelslocated in the direction showing the strong similarity (steps ST47 andST48).

Specifically, if B(D1)≦B(D2), the diagonally ascending direction isdetermined to be the direction of strong similarity, and the pixels atthe pixel positions (l−2, m+2), (l, m), (l+2, m−2) having output signalsof the same red color as the color of the pixel of interest are selectedfrom among the pixels aligned in the diagonally ascending direction(step ST47). If B(D1)>B(D2), the diagonally descending direction isdetermined to be the direction of strong similarity, and the pixels atthe pixel positions (l−2, m−2), (l, m), (l+2, m+2) having output signalsof the same red color as the color of the pixel of interest are selectedfrom among the pixels aligned in the diagonally descending direction(step ST48).

After the similarity has been thus determined, the regression analysismeans 10 receives the red output signals of the selected pixels from theframe memory 4 for use as an explanatory variable e(i) (i=1 to 3) (stepST49), and receives the blue low-pass filter outputs of the selectedpixels from low-pass filter 7 c for use as a purpose variable p(i) (i=1to 3) (step ST50).

In this case, the low-pass filter output at each pixel position isobtained by averaging the output signals of two pixels disposed ondiagonally opposite sides of the pixel position.

The regression analysis means 10 performs a regression analysis (stepST51) using the explanatory variable e(i) (red output signals) andpurpose variable p(i) (blue low-pass filter outputs), calculates theslope and intercept values of the regression line showing a correlationbetween the read and blue colors, and then calculates the generatedsignal of the blue color of the pixel of interest (step ST52), as in thefirst embodiment. The generated signal is stored into the result memory14 (step ST53).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels of interest (step ST54).

The process in step ST13 in FIG. 5 is thereby completed.

In step ST14 in FIG. 5, the above series of processes is performed as instep ST13 but by interchanging the colors red and blue to obtain redsignals at pixels having a blue output signal.

When steps ST9 to ST14 in FIG. 5 are all completed, the missing colorsof all pixels have been interpolated, so that if the output signals(pixel signals obtained by A/D conversion of the outputs of the imagingdevice 2) and the generated signals (pixel signals obtained byinterpolation) are combined, pixel signals of all colors are obtainedfor all pixels. That is, a color image filling one screen is obtained.The calculating means 9 accordingly combines the generated signalsstored in the result memory 14 with the output signals stored in theframe memory 4 and outputs them as RGB signals (step ST15).

In the second embodiment, the image similarities surrounding the pixelof interest are determined, and only the output signals of pixelsaligned in the determined direction of strong similarity are used as theexplanatory variable. As a result, for example, pixels aligned parallelto an edge contour, having a small brightness difference, are selected.Accordingly, it is possible to exclude the effect of the directionshowing the weaker similarity (the direction perpendicular to thedirection showing the stronger similarity); for example, the effect ofthe direction normal to an edge can be excluded.

Further, when a regression line is calculated in the second embodiment,instead of low-pass filter outputs, the values of output signals (pixelsignals obtained by A/D conversion of the outputs of the imaging device2) are used directly as the explanatory variable, so it is possible toobtain a regression line that more precisely describes the similarity oflocal colors. As a result, the missing colors can be generated moreprecisely.

In the second embodiment, the image similarities surrounding the pixelof interest are determined and only the output signals of pixels alignedin a determined direction of strong similarity are used as anexplanatory variable, but the output signals, instead of the low-passfilter outputs, may also be used without determining similarity. Thatis, the output signals of pixels located in all directions around thepixel of interest (without restricting the direction) may be used as theexplanatory variable.

Third Embodiment

FIG. 20 is a block diagram showing the structure of an imaging devicehaving pixel signal processing apparatus according to a third embodimentof the invention.

The third embodiment has the same structure as the second embodimentexcept for the regression analysis means 12 and calculating means 13.

In the present embodiment, pixel signals generated for a certain colorby the interpolation process including regression analysis andcalculation as described in the first and second embodiments are used asin the explanatory valuable or purpose variable in regression analysisfor interpolating another color. An example will be described below inwhich the green generated signals calculated in the second embodimentabove are used in the calculation of red and blue generated signals.

Next, the operation will be described. The interpolation procedurefollows the flowchart in FIG. 5 as in the second embodiment. The processin step ST9 obtains green generated signals at pixel positions having ared output signal, and the process in step ST10 obtains green generatedsignals at pixel positions having a blue output signal. These generatedsignals are stored in the result memory 14. The green generated signalsstored in the result memory 14 are combined with the green outputsignals stored in the frame memory 4 to obtain green pixel signals forone full screen.

In step ST11 in FIG. 5, a series of processes is carried out to obtainred signals at pixels having a green output signal. FIG. 21 is aflowchart illustrating the procedure.

First, a pixel at a pixel position (l, m) having a green output signalis selected as the pixel of interest (ST55). It will be assumed as anexample that the pixel at pixel position (3, 3) in FIG. 22 is selected.

Differing from steps ST9 and ST10, the regression analysis means 12 doesnot make a similarity determination, but instead determines whether ornot pixels of the same color (red) as the missing color of the pixel ofinterest are present on the left and right of the pixel of interest(step ST56). If they are present, the pixels at the six pixel positions(l−2, m−1), (l−2, m+1), (l, m−1), (l, m+1), (l+2, m−1), (l+2, m+1) inFIG. 22, neighboring the pixel of interest and having a red outputsignal, are selected (step ST57); otherwise, the pixels at the six pixelpositions (l−1, m−2), (l+1, m−2), (l−1, m), (l+1, m), (l−1, m+2), (l+1,m+2) in FIG. 22, neighboring the pixel of interest and having a redoutput signal, are selected (step ST58). In FIG. 22, since red pixelsare present on the left and right of the pixel of interest, the decisionresult in step ST56 is ‘Yes’, and the flow proceeds to step ST57,selecting the pixels at pixel positions (1, 2), (1, 4), (3, 2), (3, 4),(5, 2), and (5, 4).

In step ST59, if the decision in step ST56 was ‘Yes’, then the greengenerated signals g(l−2, m−1), g(l−2, m+1), g(l, m−1), g(l, m+1), g(l+2,m−1), g(l+2, m+1) at the selected pixel positions are set as anexplanatory variable e(i) (where, i=1 to 6); if the decision in stepST57 was ‘No’, then the green generated signals g(l−1, m−2), g(l+1,m−2), g(l−1, m), g(l+1, m), g(l−1, m+2), g(l+1, m+2) are set as theexplanatory variable e(i) (where, i=1 to 6). In FIG. 22, the greengenerated signals g(1, 2), g(1, 4), g(3, 2), g(3, 4), g(5, 2), g(5, 4)are set as the explanatory variable e(i) (where, i=1 to 6).

In step ST60, if the decision in step ST56 was ‘Yes’, then the redoutput signals R(l−2, m−1), R(l−2, m+1), R(l, m−1), R(l, m+1), R(l+2,m−1), R(l+2, m+1) at the selected pixel positions are set as a purposevariable p(i) (where, i=1 to 6); if the decision in step ST57 was ‘No’,then the red output signals R(l−1, m−2), R(l+1, m−2), R(l−1, m), R(l+1,m), R(l−1, m+2), R(l+1, m+2) are set as the purpose variable p(i)(where, i=1 to 6). In the example shown in FIG. 22, since the decisionin step ST57 was ‘Yes’, the red output signals R(1, 2), R(1, 4), R(3,2), R(3, 4), R(5, 2), R(5, 4) are set as the purpose variable p(i)(where, i=1 to 6).

The explanatory and purpose variables thus defined are used to perform aregression analysis, thereby calculating a regression line showing acorrelation between the green and red colors in an area neighboring thepixel of interest (step ST61). In this case, to obtain a more precisecorrelation of the local colors, among the selected pixels at the abovesix pixel positions (l−2, m−1), (l−2, m+1), (l, m−1), (l, m+1), (l+2,m−1), (l+2, m+1), or (l−1, m−2), (l+1, m−2), (l−1, m), (l+1, m), (l−1,m+2), (l+1, m+2), for example, at the six pixel positions (1, 2), (1,4), (3, 2), (3, 4), (5, 2), (5, 4) in FIG. 22, pixels having outputsignals or generated signals that differ significantly from those of theother selected pixels and the pixel of interest may be excluded from theselected pixels when the regression line is calculated.

The calculating means 13 generates the signal of the red missing colorof the pixel of interest from the calculated regression line and theoutput signal G(3, 3) of the pixel of interest (step ST62). Thegenerated signal is stored into the result memory 14 (step ST63).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels of interest (step ST64).

The process in step ST11 in FIG. 5 is thereby completed.

In step ST12 in FIG. 5, a series of processes is performed as in stepST11 above but with the color changed from red to blue to obtain bluesignals at pixels having a green output signal. This procedure is thesame as in FIG. 21, if the letter ‘R’ in steps ST55 to ST64 in FIG. 21is read as ‘B’.

First, a pixel at a pixel position (l, m) having a green output signalis selected as the pixel of interest (ST55). It will be assumed as anexample that the pixel at pixel position (3, 3) in FIG. 22 is selected.

The regression analysis means 12 determines whether or not pixels of thesame color (blue) as the missing color of the pixel of interest arepresent on the left and right of the pixel of interest (step ST56). Ifthey are present, the pixels at the six pixel positions (l−2, m−1),(l−2, m+1), (l, m−1), (l, m+1), (l+2, m−1), (l+2, m+1) in FIG. 22,neighboring the pixel of interest and having a blue output signal, areselected (step ST57); otherwise, the pixels at the six pixel positions(l−1, m−2), (l+1, m+2), (l−1, m), (l+1, m), (l−1, m+2), (l+1, m+2) inFIG. 22, neighboring the pixel of interest and having a blue outputsignal, are selected (step ST58) . In FIG. 22, since blue pixels are notpresent on the left and right of the pixel of interest, the decisionresult in step ST56 is ‘No’ and the flow proceeds to step ST58,selecting the pixels at the pixel positions (2, 1), (2, 3), (2, 5) (4,1) (4, 3), (4, 5).

In step ST59, if the decision in step ST56 was ‘Yes’, then the greengenerated signals g(l−2, m−1), g(l−2, m+1), g(l, m−1), g(l, m+1), g(l+2,m−1), g(l+2, m+1) at the selected pixel positions are set as anexplanatory variable e(i) (where, i=1 to 6); if the decision in stepST57 was ‘No’, then the green generated signals g(l−1, m−2), g(l+1,m−2), g(l−1, m), g(l+1, m), g(l−1, m+2), g(l+1, m+2) are set as theexplanatory variable e(i) (where, i=1 to 6). In FIG. 22, the greengenerated signals g(2, 1), g(2, 3), g(2, 5), g(4, 1), g(4, 3), g(4, 5)are set as the explanatory variable e(i) (where, i=1 to 6).

In step ST60, if the decision in step ST56 was ‘Yes’, then the blueoutput signals B(l−2, m−1), B(l−2, m+1), B(l, m−1), B(l, m+1), B(l+2,m−1), B(l+2, m+1) of the selected pixels are set as a purpose variablep(i) (where, i=1 to 6); if the decision in step ST57 was ‘No’, then theblue output signals B(l−1, m−2), B(l+1, m−2), B(l−1, m), B(l+1, m),B(l−1, m+2), B(l+1, m+2) are set as the purpose variable p(i) (where,i=1 to 6). In the example shown in FIG. 22, since the decision in stepST57 was ‘No’, the blue output signals B(2, 1), B(2, 3), B(2, 5), B(4,1), B(4, 3), B(4, 5) are set as the purpose variable p(i) (where, i=1 to6).

Next, the explanatory and purpose variables thus defined are used toperform a regression analysis and obtain a regression line describing acorrelation between the green and blue colors in an area neighboring thepixel of interest (step ST61). In this case, to obtain a more precisecorrelation of the local colors, among the selected pixels at the abovesix pixel positions (l−2, m−1), (l−2, m+1), (l, m−1), (l, m+1), (l+2,m−1), (l+2, m+1), or (l−1, m−2), (l+1, m−2), (l−1, m), (l+1, m), (l−1,m+2), (l+1, m+2), for example, at the six pixel positions (2, 1), (2,3), (2, 5), (4, 1), (4, 3), (4, 5) in FIG. 22, pixels having outputsignals or generated signals that are significantly different from thoseof the other selected pixels and pixel of interest may be excluded fromthe selected pixels when the regression line is calculated.

The calculating means 13 generates the signal of the missing blue colorof the pixel of interest from the calculated regression line and theoutput signal G(3, 3) of the pixel of interest (step ST62). Thegenerated signal is stored into the result memory 14 (step ST63).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels as the pixel of interest (step ST64).

The process in step ST12 in FIG. 5 is thereby completed.

In step ST13 in FIG. 5, a series of processes are carried out to obtainblue signals at pixels having a red output signal. FIG. 23 is aflowchart illustrating this procedure.

First, a pixel at a pixel position (l, m) having a red output signal isselected as the pixel of interest (ST65). It will be assumed as anexample that the pixel at pixel position (3, 3) in FIG. 24 is selected.

The regression analysis means 12 selects the blue pixels at the fourpixel positions (l−1, m−1), (l−1, m+1), (l+1, m−1), (l+1, m+1) or (2,2), (2, 4), (4, 2), (4, 4) in FIG. 24, neighboring the pixel of interestand having blue output signals, as a first group of selected pixels(first selected pixels) (step ST66).

The green generated signals g(2, 2), g(2, 4), g(4, 2), g(4, 4) at thepixel positions of the first selected pixels, stored in the resultmemory 14, are then set as part of the explanatory variable e(i) (where,i=1 to 4) (step ST67), and the blue output signals B(2, 2), B(2, 4),B(4, 2), B(4, 4) at the pixel positions of the first selected pixels areset as part of the purpose variable p(i) (where, i=1 to 4) (step ST68).

Further, the pixels at the four pixel positions (l−1, m), (l, m−1), (l,m+1), (l+1, m) or (2, 3), (3, 2), (3, 4), (4, 3) in FIG. 24, having agreen output signal, are selected as a second group of selected pixels(second selected pixels) (step ST69).

The green output signals G(2, 3), G(3, 2), G(3, 4), G(4, 3) of thesecond selected pixels are then set as another part of the explanatoryvariable e(i) (where, i=5 to 8) (step ST70), and the blue generatedsignals b(2, 3), b(3, 2), b(3, 4), b(4, 3) of the second selectedpixels, stored in the result memory 14, are set as another part of thepurpose variable p(i) (where, i=5 to 8) (step ST71).

Next, the explanatory variable e(i) (where, i=1 to 8) and purposevariable p(i) (i=1 to 8) thus defined are used to perform a regressionanalysis and obtain a regression line describing a correlation betweenthe blue and green colors in an area neighboring the pixel of interest(step ST72). In this case, to obtain a more precise correlation of thelocal colors, among the selected pixels at the above eight pixelpositions (l−1, m−1), (l−1, m+1), (l+1, m−1), (l+1, m+1), (l−1, m), (l,m−1), (l, m+1), (l+1, m), for example, the selected pixels (2, 2), (2,4), (4, 2), (4, 4), (2, 3), (3, 2), (3, 4), (4, 3) in FIG. 22, pixelshaving output signals or generated signals that differ significantlyfrom those of the other selected pixels and the pixel of interest may beexcluded from the selected pixels when the regression line iscalculated.

The calculating means 13 generates the signal of the missing blue colorof the pixel of interest from the calculated regression line and thengenerates the generated signal g(3, 3) of the pixel of interest (stepST73). The generated signal is stored into the result memory 14 (stepST74).

The above processes are repeated with different pixels of interest, thatis, by selecting different pixels as the pixel of interest (step ST75).

The process in step ST13 in FIG. 5 is thereby completed.

In step ST14 in FIG. 5, a series of processes is performed as in stepST13 above but by interchanging the red and blue colors to obtain redsignals at pixels having a blue output signal. This procedure is thesame as in FIG. 23, if the letter ‘B’ in steps ST65 to ST74 in FIG. 23is read as ‘R.’

When steps ST9 to ST14 in FIG. 5 are all completed, the missing colorsof all pixels have been interpolated, so that if the output signals andgenerated signals are combined with each other, pixel signals of allcolors are obtained for all pixels. The calculating means 9 accordinglycombines the generated signals stored in the result memory 14 with theoutput signals stored in the frame memory 4 for output as RGB signals(step ST15).

Instead of using low-pass filter outputs, steps ST11 to ST14 in thethird embodiment use the output signals of the pixels and the generatedsignals obtained by interpolation, so the pixel signals neighboring thepixel of interest can be used directly. Pixel signals in the vicinity ofthe pixel of interest can also be used. Therefore, a regression linemore precisely expressing the similarity of local colors can beobtained. As result, the missing color signals can be more preciselygenerated.

In the above first, second, and third embodiments, the two-dimensionalimaging device has been described as a device including a Bayer array ofcolor filters of the three colors red, green, and blue, but theinvention is generally applicable to any apparatus equipped with animaging device with N types of photoelectric conversion elements, eachhaving one of a first spectral characteristic to an N-th spectralcharacteristic (in the above embodiments, N=3), arrayed on atwo-dimensional plane.

In the first, second, and third embodiment, the regression analysismeans (8, 10, or 12) receives pixel signals having a K-th (in the aboveexamples, K-th denotes red, green, or blue) spectral characteristic andpixel signals having an L-th spectral characteristic (in the aboveexamples, L denotes red, green, or blue but L differs from K) at aplurality of pixel positions in an area neighboring a first pixelposition of interest, and performs a regression analysis using the pixelsignals having the K-th spectral characteristic as an explanatoryvariable and the pixel signals having the L-th spectral characteristicas a purpose variable to calculate a regression equation expressing acorrelation of the pixel signals having the K-th spectral characteristicwith the pixel signals having the L-th spectral characteristic. Thecalculating means (9 or 13) determines the pixel signal having the L-thspectral characteristic at the first pixel position of interest byapplying a conversion formula based on the above regression equation tothe pixel signal having the K-th spectral characteristic at the firstpixel position of interest.

In the first, second, and third embodiment, the low-pass filters (7 a to7 c) perform low-pass filtering on the output signals from the N typesof photoelectric conversion elements, and generate low-pass filteroutputs corresponding to the first to N-th spectral characteristics.

In the first embodiment, the pixel signals obtained by the low-passfiltering are used as both explanatory and purpose variables; in thesecond embodiment, the output signals are used as an explanatoryvariable and the pixel signals obtained by the low-pass filtering areused as a purpose variable.

In the third embodiment, a certain color pixel signal at a certainpixel, generated by the same regression analysis and calculation as inthe second embodiment, is used as part of the explanatory variable in aregression analysis for interpolating a pixel signal of another color atanother pixel in the neighborhood of the certain pixel.

Instead, a certain color pixel signal at a certain pixel, generated bythe same regression analysis and calculation as in the first embodiment,may also be used as the explanatory variable in the regression analysisfor interpolating the pixel signal of the other color at another pixelin the neighborhood of the certain pixel.

Further, instead of the regression analyses and calculations used in thefirst and second embodiments, known interpolating means may also beused. Those interpolating means can be generalized as follows: a pixelsignal having the K-th spectral characteristic at a second pixelposition of interest, which is located in an area neighboring a firstpixel of interest and where there is a pixel signal having an M-thspectral characteristic (M being an integer from 1 to N different fromK), is obtained by interpolation based on pixel signals having the M-thspectral characteristic and pixel signals having the K-th spectralcharacteristic at a plurality of pixel positions in an area neighboringthe second pixel position of interest.

A structure in which the interpolation means performs interpolationusing the same regression analyses and calculations as in the first andsecond embodiments can be generally described as follows. That is, theregression analysis means receives pixel signals having the M-thspectral characteristic and pixel signals having the K-th spectralcharacteristic at a plurality of pixel positions in an area neighboringthe second pixel position of interest, and performs a regressionanalysis using the pixel signals having the M-th spectral characteristicas an explanatory variable and the pixel signals having the K-thspectral characteristic as a purpose variable to calculate a regressionequation expressing a correlation of the pixel signals having the M-thspectral characteristic with the pixel signals having the K-th spectralcharacteristic. The calculating means then determines the pixel signalhaving the K-th spectral characteristic at the second pixel position ofinterest by applying a conversion formula based on the regressionequation to the pixel signal having the M-th spectral characteristic atthe second pixel position of interest. The pixel signal thus obtained asa pixel signal obtained by interpolation and having the K-th spectralcharacteristic is used as an explanatory variable for interpolating apixel signal having an L-th spectral characteristic.

The third embodiment further uses pixel signals obtained by theregression analysis and calculation as part of the purpose variable in aregression analysis for interpolating a pixel signal of another color.This is generalized to the case where colors have first to N-th spectralcharacteristics as described below.

The apparatus has further interpolation means that obtains a pixelsignal having an L-th spectral characteristic at a second pixel positionof interest by interpolation, based on pixel signals having an M-thspectral characteristic (M being an integer from 1 to N different fromL) and pixel signals having the L-th spectral characteristic at aplurality of pixel positions in an area neighboring the second pixelposition of interest. In addition, the regression analysis means usesthe pixel signals obtained by the above interpolation as part of thepurpose variable.

A structure in which the interpolation means comprises the regressionanalysis means and calculating means described in the above thirdembodiment can be generally described as follows. That is, theregression analysis means receives pixel signals having an M-th spectralcharacteristic and pixel signals having an L-th spectral characteristicat a plurality of pixel positions in an area neighboring a second pixelposition of interest, and performs a regression analysis using the pixelsignals having the M-th spectral characteristic as an explanatoryvariable and the pixel signals having the L-th spectral characteristicas a purpose variable to calculate a regression equation expressing acorrelation of the pixel signals having the M-th spectral characteristicwith the pixel signals having the L-th spectral characteristic. Thecalculating means then determines the pixel signal having the L-thspectral characteristic at the second pixel position of interest byapplying a conversion formula based on the regression equation to thepixel signal having the M-th spectral characteristic at the second pixelposition of interest. The pixel signal thus obtained as a pixel signalobtained by interpolation and having the L-th spectral characteristic isused as part of the purpose variable.

In the first to third embodiments, the regression means (8, 10, or 12)selects a line represented by the equation y=a·x+b (‘y’ being a pixelsignal with a Y-th spectral characteristic, ‘x’ being a pixel signalwith an X-th spectral characteristic, and ‘a’ and ‘b’ being constants)as a regression equation. The calculating means (9 or 13) obtains apixel signal Y′ having the Y-th spectral characteristic at a pixelposition of interest by substituting a pixel signal X having the X-thspectral characteristic at the pixel position of interest into theconversion formula Y′=a·X+b based on the above line. When a pixel signalhaving the L-th spectral characteristic is generated at a first pixelposition of interest where there is a pixel signal having the K-thspectral characteristic, X=K, Y=L, and Y′=L′. When a pixel signal havingthe K-th spectral characteristic is generated at a second pixel positionof interest where there is a pixel signal having the M-th spectralcharacteristic, X=M, Y=K, and Y′=K′. When a pixel signal having the L-thspectral characteristic is generated at a second pixel position ofinterest where there is a pixel signal having the M-th spectralcharacteristic, X=M, Y=L, and Y′=L′.

In step ST2 in FIG. 8, the parameters k1, k2, k3 are obtained from anexplanatory variable e(i) and a purpose variable p(i), but when theexplanatory variable is denoted x(i) and the purpose variable is denotedy(i) as described above, they are obtainable by the following equation.$\begin{matrix}{{k\quad 1} = {N \cdot {\sum\limits_{i = 1}^{N}\left( {{x(i)} \cdot {y(i)}} \right)}}} \\{{k\quad 2} = {\sum\limits_{i = 1}^{N}{{x(i)} \cdot {\sum\limits_{i = 1}^{N}{y(i)}}}}} \\{{{k\quad 3} = {{N \cdot {\sum\limits_{i = 1}^{N}{x(i)}^{2}}} - \left( {\sum\limits_{i = 1}^{N}{y(i)}} \right)^{2}}}\left( {N\text{:}\quad{data}\quad{count}} \right)}\end{matrix}$

1-20. (canceled)
 21. A pixel signal processing apparatus that, given agroup of pixel signals from pixels arrayed on a two-dimensional plane,each pixel having one of a first to an N-th spectral characteristic,generates a pixel signal having an L-th spectral characteristic at afirst pixel position of interest where there is a pixel signal having aK-th spectral characteristic (K and L being different integers between 1and N, inclusive), comprising: a regression analysis means forperforming a regression analysis in a plurality of pixel positions in anarea neighboring the first pixel position of interest, using the pixelsignals having the K-th spectral characteristic as an explanatoryvariable and the pixel signals having the L-th spectral characteristicas a purpose variable, to calculate a regression liney=a·x+b   (1) (‘y’ being the pixel signal having the L-th spectralcharacteristic, ‘x’ being the pixel signals having the L-th spectralcharacteristic, ‘a’ and ‘b’ being constants representing the slope andintercept of the regression line) expressing a correlation of the pixelsignals having the K-th spectral characteristic with the pixel signalshaving the L-th spectral characteristic; a calculating means fordetermining the pixel signal having the L-th spectral characteristic atthe first pixel position of interest by applying a conversion formulabased on the regression line to the pixel signal having the K-thspectral characteristic at the first pixel position of interest; and aselection means for sequentially selecting different pixels as the pixelof interest and, for each selected pixel of interest, using theregression analysis means and the calculating means to determine thepixel signal having the L-th spectral characteristic.
 22. The pixelsignal processing apparatus of claim 21, further comprising an imagingdevice with N types of photoelectric conversion elements, each havingone of the first to N-th spectral characteristics, arrayed on atwo-dimensional plane, wherein: the selection means determines the K-thand L-th spectral characteristics in order of the strength of thecorrelation between their spectral characteristics.
 23. The pixel signalprocessing apparatus of claim 22, wherein: the imaging device has one ofred (R), green (G), and blue (B) spectral characteristics; the selectionmeans first determines the green pixel signals at pixel positions wherered pixel signals are present and the green pixel signals at pixelpositions where blue pixel signals are present; next, the selectionmeans determines the red pixel signals at pixel positions where greenpixel signals are present and the blue pixel signals at pixel positionswhere green pixel signals are present; and finally, the selection meansfirst determines the blue pixel signals at pixel positions where redpixel signals are present and the red pixel signals at pixel positionswhere blue pixel signals are present.
 24. A pixel signal processingmethod that, given a group of pixel signals from pixels arrayed on atwo-dimensional plane, each pixel having one of a first to an N-thspectral characteristic, generates a pixel signal having an L-thspectral characteristic at a first pixel position of interest wherethere is a pixel signal having a K-th spectral characteristic (K and Lbeing different integers between 1 and N, inclusive), comprising: aregression analysis step for performing a regression analysis in aplurality of pixel positions in an area neighboring the first pixelposition of interest, using the pixel signals having the K-th spectralcharacteristic as an explanatory variable and the pixel signals havingthe L-th spectral characteristic as a purpose variable, to calculate aregression liney=a·x+b   (1) (‘y’ being the pixel signal having the L-th spectralcharacteristic, ‘x’ being the pixel signals having the L-th spectralcharacteristic, ‘a’ and ‘b’ being constants representing the slope andintercept of the regression line) expressing a correlation of the pixelsignals having the K-th spectral characteristic with the pixel signalshaving the L-th spectral characteristic; a calculating step fordetermining the pixel signal having the L-th spectral characteristic atthe first pixel position of interest by applying a conversion formulabased on the regression line to the pixel signal having the K-thspectral characteristic at the first pixel position of interest; and aselection step for sequentially selecting different pixels as the pixelof interest and, for each selected pixel of interest, using theregression analysis step and the calculating step to determine the pixelsignal having the L-th spectral characteristic.
 25. The pixel signalprocessing method of claim 24, wherein said pixel signals are associatedwith an imaging device with N types of photoelectric conversionelements, each having one of the first to N-th spectral characteristics,arrayed on a two-dimensional plane, and wherein: the selection stepdetermines the K-th and L-th spectral characteristics in order of thestrength of the correlation between their spectral characteristics. 26.The pixel signal processing method of claim 25, wherein: eachphotoelectric conversion element has one of red (R), green (G), and blue(B) spectral characteristics; the selection step first determines thegreen pixel signals at pixel positions where red pixel signals arepresent and the green pixel signals at pixel positions where blue pixelsignals are present; next, the selection step determines the red pixelsignals at pixel positions where green pixel signals are present and theblue pixel signals at pixel positions where green pixel signals arepresent; and finally, the selection step first determines the blue pixelsignals at pixel positions where red pixel signals are present and thered pixel signals at pixel positions where blue pixel signals arepresent.